Composite imaging method and system

ABSTRACT

In a first aspect of the invention, a method is provided for forming a group image. In accordance with the method, a set of imaging information is obtained depicting a scene over a period of time. Elements in the set of imaging information are distinguished and attributes of the elements in the set of image information are examined. Imaging information is selected from the set of imaging information depicting each element with the selection being made according to the attributes for that element. A group image is formed based upon the set of imaging information with the archival image incorporating the selected image information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a divisional of application Ser. No. 10/431,057, filed May 7,2003.

FIELD OF THE INVENTION

The present invention relates to imaging systems and more particularlyto imaging systems that are adapted to form images having multipleobjects therein.

BACKGROUND OF THE INVENTION

Professional and amateur photographers often capture images of groups ofpeople such as images of families and athletic teams. Such group imagesare typically used for commemorative purposes. A common problem withsuch group images is that often one or more members of the group willhave an appearance at the time that the group image is captured that themember does not prefer. For example, group members can blink, look away,make a comment or otherwise compose their facial attributes in anon-preferable way. Similar problems can occur whenever images arecaptured that include more than one element. Examples of such elementsinclude people, as described above, animals, objects, areas such as abackground of a scene, and/or any other photographic subject that canchange over time. Typically, photographers address this problem bycapturing multiple images of the group of elements and selecting fromthe multiple images a group image that shows all of the elements in thegroup image having a generally acceptable appearance. Even where this isdone, it is often the case that one or more elements has a less thanoptimal appearance.

Various forms of image editing software can be used to attempt toimprove the appearance of elements in a group image. Such editingsoftware typically includes automatic image correction algorithms thatcan resolve common image problems such as the so-called red-eye problemthat can occur in images of people. See, for example, commonly assignedU.S. Patent Application Publication No. 2003-0053663 entitled “Methodand Computer Program Products for Locating Facial Features” filed byChen et al. on Nov. 26, 2001. Further, advanced users of such imageediting software can use manual image editing tools such as AdobePhotoShop™ software sold by Adobe Systems Inc., San Jose, Calif., USA,to manually alter images. It will be appreciated however, that the useof such image editing tools to correct a group image is time consumingand can yield results that have a less than authentic appearance. Whatis needed therefore is an imaging system and method that can effectivelyform optimal group images with an authentic appearance in a less timeconsuming manner.

SUMMARY OF THE INVENTION

In a first aspect of the invention, a method is provided for forming agroup image. In accordance with the method, a set of imaging informationis obtained depicting a scene over a period of time. Elements in the setof imaging information are distinguished and attributes of the elementsin the set of image information are examined. Imaging information isselected from the set of imaging information depicting each element withthe selection being made according to the attributes for that element. Agroup image is formed based upon the set of imaging information with thearchival image incorporating the selected image information.

In another aspect of the invention, a method for forming an image isprovided. In accordance with this method, a set of imaging informationis obtained depicting a scene over a period of time. A base image isprovided based on the set of image information. Elements are identifiedin the base image and portions of the set of imaging informationdepicting each of the elements are ordered according to an attribute ofeach element. Imaging information from the set of imaging information isselected depicting each element according to the ordering. An image isformed based upon the set of imaging information with the base imageincorporating the selected image information.

In still another aspect of the invention, a method for forming an imageis provided. In accordance with this method, images of a scene areobtained at different times. Elements in the images are identified.Attributes for each of the elements in each of the images are determinedand it is determined for each element which image shows the elementhaving preferred attributes. An image is prepared of the scene with eachelement having an appearance that corresponds to the appearance of theelement in the image that shows the preferred attributes for theelement.

In another aspect of the invention, a computer program product isprovided having data stored thereon for causing an imaging system toperform a method for forming a group image. In accordance with themethod, a set of imaging information is obtained depicting a scene overa period of time. Elements in the set of imaging information aredistinguished and attributes of the elements in the set of imageinformation are examined. Imaging information is selected from the setof imaging information depicting each element with the selection beingmade according to the attributes for that element. A group image isformed based upon the set of imaging information with the archival imageincorporating the selected image information.

In another aspect of the invention, a computer program product isprovided having data stored thereon for causing the imaging system toperform a method for forming an image. In accordance with this method, aset of imaging information is obtained depicting a scene over a periodof time. A base image is provided based on the set of image information.Elements are identified in the base image and portions of the set ofimaging information depicting each of the elements are ordered accordingto an attribute of each element. Imaging information from the set ofimaging information is selected depicting each element according to theordering. An image is formed based upon the set of imaging informationwith the base image incorporating the selected image information.

In another aspect of the invention, a computer program product isprovided having data stored thereon for causing imaging system toperform a method for forming a group image. In accordance with thismethod, images of a scene are obtained at different times. Elements inthe images are identified. Attributes for each of the elements in eachof the images are determined and it is determined for each element whichimage shows the element having preferred attributes. An image isprepared of the scene with each element having an appearance thatcorresponds to the appearance of the element in the image that shows thepreferred attributes for the element.

In another aspect of the invention, an imaging system is provided. Theimaging system has a source of a set of image information and a signalprocessor adapted to receive the set of image information identified, toidentify elements in the set of image information, to distinguishelements in the set of image information, and to examine the attributesof the elements in the set of image information. Wherein the signalprocessor is further adapted to select imaging information from the setof imaging information; to taking each element, with the selection beingmade according to the attributes for that element; and, to form a groupimage based upon the set of imaging information with the group imageincorporating the selected image information.

In still another aspect of the invention, an imaging system is provided.In accordance with this aspect, the imaging system has a source ofimaging information and a signal processor adapted to obtain a set ofimaging information from the source of imaging information depicting ascene over a period of time. The signal processor provides a base imagebased upon the set of imaging information and identifies elements in thebase image. The signal processor orders the portions of the set ofimaging information depicting each of the elements according to anattribute of each element. The processor selects imaging informationfrom the set of imaging information depicting each element according tothe ordering and forms a group image incorporating the selected imageinformation.

In accordance with a further embodiment aspect of the invention, animaging system is provided comprising a source of images of a scenecaptured at different times. The signal processor is adapted to obtainimages from the source, to identify elements in the images, to determineattributes for each of the elements in each of the images and todetermine for each element which image shows an element having preferredattributes wherein the signal processor prepares an image of the scenewith each element having an appearance that corresponds to theappearance of the element in the image that shows the preferredattributes for the element.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows one embodiment of a composite imaging system of the presentinvention;

FIG. 2 shows a back view of the embodiment of FIG. 1;

FIG. 3 shows a flow diagram of one embodiment of a method for forming agroup image in accordance with the present invention;

FIG. 4 shows an illustration of objects and elements within an image;

FIG. 5 shows an illustration of the use of a set of image information toform an image;

FIG. 6 shows an illustration of the use of a desired facial expressionor mood selection to influence the appearance of an image;

FIG. 7 shows a flow diagram of one embodiment of a method for approvingand ordering an image in accordance with the present invention; and

FIG. 8 shows an illustration depicting the operation of anotherembodiment of the method of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a block diagram of one embodiment of an imaging system 10.FIG. 2 shows a top, back, right side perspective view of the imagingsystem 10 of FIG. 1. As is shown in FIGS. 1 and 2, imaging system 10comprises a body 20 containing an image capture system 22 having a lenssystem 23, an image sensor 24, a signal processor 26, an optionaldisplay driver 28 and a display 30. In operation, light from a scene isfocused by lens system 23 to form an image on image sensor 24. Lenssystem 23 can have one or more elements. Lens system 23 can be of afixed focus type or can be manually or automatically adjustable. Lenssystem 23 is optionally adjustable to provide a variable zoom that canbe varied manually or automatically. Other known arrangements can beused for lens system 23.

Light from the scene that is focused by lens system 23 onto image sensor24 is converted into image signals I representing an image of the scene.Image sensor 24 can comprise a charge couple device (CCD), acomplimentary metal oxide sensor (CMOS), or any other electronic imagesensor known to those of ordinary skill in the art. Image signals I canbe in digital or analog form.

Signal processor 26 receives image signals I from image sensor 24 andtransforms image signal I into a set of imaging information S. Set ofimage information S can comprise a set of still images or other imageinformation in the form of a video stream of apparently moving images.In such embodiments, the set of image information S can comprise imageinformation in an interleaved or interlaced image form. Signal processor26 can also apply image processing algorithms to image signals I in theformation of the set of image information S. These can include but arenot limited to color and exposure balancing, interpolation andcompression. Where image signals I are in the form of analog signals,signal processor 26 converts these analog signals into a digital form.

A controller 32 controls the operation of image capture system 22,including lens system 23, image sensor 24, signal processor 26, and amemory such as memory 40 during imaging operations. Controller 32 causesimage sensor 24, signal processor 26, display 30 and memory 40 tocapture, store and display images in response to signals received from auser input system 34, data from signal processor 26 and data receivedfrom optional sensors 36. Controller 32 can comprise a microprocessorsuch as a programmable general purpose microprocessor, a dedicatedmicro-processor or micro-controller, or any other system that can beused to control operation of imaging system 10.

User input system 34 can comprise any form of transducer or other devicecapable of receiving an input from a user and converting this input intoa form that can be used by controller 32 in operating imaging system 10.For example, user input system 34 can comprise a touch screen input, a4-way switch, a 6-way switch, an 8-way switch, a stylus system, atrackball system, a joystick system, a voice recognition system, agesture recognition system or other such systems. In the embodimentshown in FIGS. 1 and 2 user input system 34 includes a shutter triggerbutton 60 that sends a trigger signal to controller 32 indicating adesire to capture an image.

As shown in FIGS. 1 and 2, user input system 34 also includes awide-angle zoom button 62, and a tele zoom button 64 that controls thezoom settings of lens system 23 causing lens system 23 to zoom out whenwide angle zoom button 62 is depressed and to zoom out when tele zoombutton 64 is depressed. Wide-angle zoom lens button 62 and telephotozoom button 64 can also be used to provide signals that cause signalprocessor 26 to process image signal I to provide a set of imageinformation that appears to have been captured at a different zoomsetting than that actually provided by the optical lens system. This canbe done by using a subset of the image signal I and interpolating asubset of the image signal I to form the set of image information S.User input system 34 can also include other buttons including the Fix-Itbutton 66 shown in FIG. 2 and the Select-It button 68 shown in FIG. 2,the function of which will be described in greater detail below.

Sensors 36 are optional and can include light sensors, range finders andother sensors known in the art that can be used to detect conditions inthe environment surrounding imaging system 10 and to convert thisinformation into a form that can be used by controller 32 in governingoperation of imaging system 10.

Controller 32 causes a set of image information S to be captured when atrigger condition is detected. Typically, the trigger condition occurswhen a user depresses shutter trigger button 60, however, controller 32can determine that a trigger condition exists at a particular time, orat a particular time after shutter trigger button 60 is depressed.Alternatively, controller 32 can determine that a trigger conditionexists when optional sensors 36 detect certain environmental conditions.

Controller 32 can also be used to generate metadata M in associationwith each image. Metadata M is data that is related to a set of imageinformation or a portion of set of image information S but that is notnecessarily observable in the image data itself. In this regard,controller 32 can receive signals from signal processor 26, camera userinput system 34 and other sensors 36 and, optionally, generates metadataM based upon such signals. Metadata M can include but is not limited toinformation such as the time, date and location that the archival imagewas captured, the type of image sensor 24, mode setting information,integration time information, taking lens unit setting information thatcharacterizes the process used to capture the archival image andprocesses, methods and algorithms used by imaging system 10 to form thearchival image. Metadata M can also include but is not limited to anyother information determined by controller 32 or stored in any memory inimaging system 10 such as information that identifies imaging system 10,and/or instructions for rendering or otherwise processing the archivalimage with which metadata M is associated that can also be incorporatedinto the image metadata such an instruction to incorporate a particularmessage into the image. Metadata M can further include image informationsuch as a set of display data, a set of image information S or any partthereof. Metadata M can also include any other information entered intoimaging system 10.

Set of image information S and optional metadata M, can be stored in acompressed form. For example where set of image information S comprisesa sequence of still images, the still images can be stored in acompressed form such as by using the JPEG (Joint Photographic ExpertsGroup) ISO 10918-1 (ITU-T.81) standard. This JPEG compressed image datais stored using the so-called “Exif” image format defined in theExchangeable Image File Format version 2.2 published by the JapanElectronics and Information Technology Industries Association JEITACP-3451. Similarly, other compression systems such as the MPEG-4 (MotionPictures Export Group) or Apple Quicktime™ standard can be used to storea set of image information that is received in a video form. Other imagecompression and storage forms can be used.

The set of image information S can be stored in a memory such as memory40. Memory 40 can include conventional memory devices including solidstate, magnetic, optical or other data storage devices. Memory 40 can befixed within imaging system 10 or it can be removable. In the embodimentof FIG. 1, imaging system 10 is shown having a memory card slot 46 thatholds a removable memory 48 such as a removable memory card and has aremovable memory interface 50 for communicating with removable memory48. The set of image information can also be stored in a remote memorysystem 52 that is external to imaging system 10 such as a personalcomputer, computer network or other imaging system.

In the embodiment shown in FIGS. 1 and 2, imaging system 10 has acommunication module 54 for communicating with the remote memory system.The communication module 54 can be for example, an optical, radiofrequency or other transducer that converts image and other data into aform that can be conveyed to the remote imaging system by way of anoptical signal, radio frequency signal or other form of signal.Communication module 54 can also be used to receive a set of imageinformation and other information from a host computer or network (notshown). Controller 32 can also receive information and instructions fromsignals received by communication module 54 including but not limitedto, signals from a remote control device (not shown) such as a remotetrigger button (not shown) and can operate imaging system 10 inaccordance with such signals.

Signal processor 26 optionally also converts image signals I into a setof display data DD that is in a format that is appropriate forpresentation on display 30. Display 30 can comprise, for example, acolor liquid crystal display (LCD), organic light emitting display(OLED) also known as an organic electroluminescent display (OELD) orother type of video display. Display 30 can be external as is shown inFIG. 2, or it can be internal for example used in a viewfinder system38. Alternatively, imaging system 10 can have more than one displaywith, for example, one being external and one internal.

Typically, display 30 has less imaging resolution than image sensor 24.Accordingly, signal processor 26 reduces the resolution of image signalI when forming the set of display data DD adapted for presentation ondisplay 30. Down sampling and other conventional techniques for reducingthe overall imaging resolution can be used. For example, resamplingtechniques such as are described in commonly assigned U.S. Pat. No.5,164,831 “Electronic Still Camera Providing Multi-Format Storage OfFull And Reduced Resolution Images” filed by Kuchta et al., on Mar. 15,1990, can be used. The set of display data DD can optionally be storedin a memory such as memory 40. The set of display data DD can be adaptedto be provided to an optional display driver 28 that can be used todrive display 30. Alternatively, the display data can be converted intosignals that can be transmitted by signal processor 26 in a form thatdirectly causes display 30 to present a set of display data DD. Wherethis is done, display driver 28 can be omitted.

Imaging system 10 can obtain a set of image information in a variety ofways. For example, imaging system 10 can capture a set of imageinformation S using image sensor 24. Imaging operations that can be usedto obtain a set of image information S from image capture system 22include a capture process and can optionally also include a compositionprocess and a verification process.

During the optional composition process, controller 32 causes signalprocessor 26 to cooperate with image sensor 24 to capture image signalsI and present a set of display data DD on display 30. In the embodimentshown in FIGS. 1 and 2, controller 32 enters the image composition phasewhen shutter trigger button 60 is moved to a half depression position.However, other methods for determining when to enter a composition phasecan be used. For example, one of user input system 34, for example, the“fix-it” button 66 shown in FIG. 2 can be depressed by a user of imagingsystem 10, and can be interpreted by controller 32 as an instruction toenter the composition phase. The set of display data DD presented duringcomposition can help a user to compose the scene for the capture of aset of image information S.

The capture process is executed in response to controller 32 determiningthat a trigger condition exists. In the embodiment of FIGS. 1 and 2, atrigger signal is generated when shutter trigger button 60 is moved to afull depression condition and controller 32 determines that a triggercondition exists when controller 32 detects the trigger signal. Duringthe capture process, controller 32 sends a capture signal causingdigital signal processor 26 to obtain image signals I and to process theimage signals I to form a set of image information S. A set of displaydata DD corresponding the set of image information S is optionallyformed for presentation on display 30.

During the verification phase, the corresponding display data DD issupplied to display 30 and is presented for a period of time. Thispermits a user to verify that the set of image information S isacceptable. In one alternative embodiment, signal processor 26 convertseach image signal I into the set of imaging information S and thenmodifies the set of imaging information S to form a set of display dataDD.

The group image forming features of imaging system 10 of FIGS. 1 and 2will now be described with reference to FIGS. 3, 4, 5, 6, 7, and 8. FIG.3 shows a flow diagram of an embodiment of a method for composing animage. FIG. 4 shows an illustration of objects and elements within animage. FIG. 5 shows an illustration of the use of a set of imageinformation to form an image. FIG. 6 shows an illustration of the use ofa desired facial expression or mood selection to influence theappearance of an image. FIG. 7 shows a flow diagram of an embodiment ofa method for approving and ordering an image. FIG. 8 shows anillustration depicting the operation of another embodiment of the methodof the present invention. In the following description, a method will bedescribed. However, in another embodiment, the methods describedhereinafter can take the form of a computer program product for forminga group image.

The computer program product for performing the described methods can bestored in a computer readable storage medium. This medium may comprise,for example: magnetic storage media such as a magnetic disk (such as ahard drive or a floppy disk) or magnetic tape; optical storage mediasuch as an optical disc, optical tape, or machine readable bar code;solid state electronic storage devices such as random access memory(RAM), or read only memory (ROM); or any other physical device or mediumemployed to store a computer program. The computer program product forperforming the described methods may also be stored on a computerreadable storage medium that is connected to imaging system 10 by way ofthe Internet or other communication medium. Those skilled in the artwill readily recognize that the equivalent of such a computer programproduct can also be constructed in hardware.

In describing the following methods, it should be apparent that thecomputer program product embodiment can be utilized by any well-knowncomputer system, including but not limited to the computing systemsincorporated in imaging system 10 described above. However, many othertypes of computer systems can be used to execute the computer programembodiment. Examples of such other computer systems include personalcomputers, personal digital assistants, work station, Internetapplications and the like. Consequently, the computer system will not bediscussed in further detail herein.

Turning now to FIG. 3, the method of forming a group image begins withimaging system 10 entering a mode for forming a group image (step 70).The group image forming mode can be entered automatically withcontroller 32 entering the mode as a part of an initial start upoperation that is executed when imaging system 10 is activated.Alternatively, the group image mode can be entered automatically whensignal processor 26 determines that a set of image information containsan arrangement of image elements that suggests that the scene can bebeneficially processed using a set of image information. The group imagemode can also be entered when controller 32 detects a user selection atuser input system 34 such as striking Fix-It button 66 shown in FIG. 2.

A set of image information S is then obtained (step 72). As is describedabove, the set of imaging information S can be obtained using theimaging operations described above. For example, controller 32 can beadapted to receive a trigger signal from user input 34. When the triggersignal is received, controller 32 causes a set of image information S tobe obtained from image sensor 24 depicting the scene over a period oftime. This set of image information S can comprise, for example, asequence of archival still images captured over the period of time. Thisset of image information S can also comprise interlaced or other formsof video image information captured over the period of time. The periodof time can begin at the moment that the trigger condition is detected.

Alternatively, where an image composition phase is used to captureimages, controller 32 can cause a set of image information S to bestored in a first in first out buffer in a memory such as memory 40during composition. Where this is done, the set of image information Scan be captured during composition and fed into the buffer, so that atthe time controller 32 determines that trigger condition exists, thebuffer contains imaging information depicting the scene for a period oftime prior to the point in time where controller 32 determines that thetrigger condition exists. In this way, the set of imaging information Sobtained can include imaging information obtained prior to the detectedtrigger condition. In another alternative embodiment of the presentinvention, a set of imaging information S can be obtained from anymemory in imaging system 10. For example, the set of imaging informationS can be obtained from a removable memory 48 having the set of imaginginformation recorded therein by another image capture device (notshown). Further, the set of imaging information can be obtained from anexternal source by way of communication module 54.

Objects and elements within a base image are distinguished within theset of imaging information (step 74). In the embodiment shown, this canbe done by selecting a base image from the image stream and identifyingobjects and elements within the base image. The base image can beselected by selecting the first image in the set of imaging informationS, automatically selecting an image that corresponds to scene conditionsat the time that the trigger condition is detected, or automaticallyselecting any other image in the set of imaging information S based onsome other selection strategy. The base image contains objects such as aregion, person, place, or thing. An object can contain multiple elementsfor example, where the object is a face of a person, elements cancomprise the eyes and mouth of the person. Objects and/or elements canbe detected in the imaging information using a variety of detectionalgorithms and methods including but not limited to human body detectionalgorithms such as those disclosed in commonly assigned U.S. PatentApplication Publication No. 2002/0076100 entitled “Image processingmethod for detecting human figures in a digital image” filed by Luo onDec. 14, 2000, and human face recognition algorithms such as thosedescribed in commonly assigned U.S. Patent Application Publication No.2003/0021448 entitled “Method For Detecting Eye and Mouth Positions in aDigital Image” filed by Chen et al. on May 1, 2001.

It will be appreciated that the step of sorting the image for objectscan simplify the process of distinguishing elements within the objectsby reducing the set of elements that are likely to be within certainareas of the image. However, this is optional and elements can also beidentified in a base image without first distinguishing objects.Further, in certain embodiments, objects and/or elements can bedistinguished within the set of imaging information S without firstforming a base image.

Attributes of each of the elements are then examined (step 76). Eachelement has variable attributes that can change over the period of timecaptured in the set of image information. For example, the eyes of aface can open and close during the period of time, or a mouth can shiftfrom smiling to not smiling. Time variable attributes of elements suchas eyes or a mouth can be identified automatically, as they are easilyrecognizable as being of interest in facial images. However, in certaincircumstances the user of imaging system 10 can identify, manually,elements and attributes of interest. Where this is to be done, the baseimage is presented on display 30 and the user of imaging system 10 canuse user input system 34 to identify objects, elements and attributes inthe base image that are of interest. Such objects, elements andattributes can be identified for example by name, icon, image, outline,arrow, or other visual or audio symbol or signal. For convenience, theidentifier used for the element can be presented on display 30.

FIG. 4 shows a drawing of one example of a base image 90 from a set ofimage information, its objects, elements, and attributes. Image 90 iscomposed of two objects: face 92 and face 94. Face 92 is composed of twoelements: eye element 96 and mouth element 98. Face 94 is likewisecomposed of two elements: eye element 100 and mouth element 102. Theattribute of eye element 96 is eyes open. The attribute of eye element100 is eyes closed. The attribute of mouth element 98 is mouth notsmiling. The attribute of mouth element 102 is mouth smiling.

Objects and elements distinguished in base image 90 are thendistinguished in the other images over the set of imaging information Sin like manner (step 78). Attributes of each of the elements are thendetermined in the remaining portions of the set of imaging information S(step 80).

The imaging information depicting each element in the set of imaginginformation is ordered in decreasing attribute level across theavailable imaging information in the set of imaging information (step82). This ordering is performed by comparing the appearance of each ofthe elements in the stream of imaging information to preferredattributes for the element. For example, if the best group image of agroup of people is desired, then the attributes of eyes open and mouthsmiling are of high priority. Therefore the imaging informationassociated with the element can be ordered based upon which imaginginformation depicts an eye element having the attribute of an open eyeat the top of an ordered list and imaging information depicting the eyeelement having a closed or partially closed eye at the bottom of theordered list. Similarly, the ordered list of the element of a mouthhaving the attribute of smiling would be at the top of an ordered listfor the mouth element and imaging information depicting a mouth elementhaving a non smiling arrangement would be at the bottom of the orderedlist.

The preferred attributes used for the ordering scheme can be determinedautomatically by analysis of the set of imaging information S todetermine what attributes can be preferred for the image. Alternatively,the attributes to be used for ordering the image can be set by the user,for example by using user input system 34. Other imaging informationordering criteria can be used where other subjects are involved. Forexample, where the objects in the image include a group of show dogsposing for an image, while doing a similar acrobatic activity such asjumping, the head direction of each dog can be determined and apreference can be shown for the attribute of each dog facing in thegeneral direction of the camera.

FIG. 5 illustrates an example of the operation of the method of FIG. 3.FIG. 5 shows a set of imaging information 110 comprising a sequence offour images 112, 114, 116, 118 captured over a period of time. As isshown, each of the images 112, 114, 116 and 118 contains two objects,i.e., faces 120 and 122. The eye elements 124 and 126, and mouthelements 128 and 130 are identified as important image elements. Theattributes of eye elements 124 and 126 in image 112 are examined firstand then the attributes of eye elements 124 and 126 in images 114, 116,and 118 are examined. The preferred element attributes are thenautomatically determined: in this case open eyes and smiling mouth.Elements Images 124 128 126 130 112 medium high high medium 114 mediumhigh medium high 116 medium medium low high 118 high medium medium high

An ordered list of imaging information depicting eye attributes 124 and126 is formed based upon closeness of the attributes of the eye elementsin each of images 112, 114, 116 and 118 to the preferred attributes ofthe eye elements. See Table 2. The imaging information depicting the eyeelement having highest ordered attributes on the ordered list is used toform a group image 132. Similarly, the mouth elements 128 and 130 inimages 112 are examined and compared to the mouth elements 128 and 130in images 114, 116 and 118, and an ordered list of imaging informationhaving preferred mouth attributes is determined. See Table 2. The mouthelements 128 and 130 that are highest on the ordered list of mouthattributes are used to form the group image 132. Elements 124 128 126130 Priority 118 112 112 114 112 114 114 116 114 116 118 118 116 118 116112

Other strategems can also be used in forming an ordered list of imaginginformation.

The group image is then automatically composed (step 84). This can bedone in a variety of ways. In one embodiment, controller 32 and signalprocessor 26 select an interim image for use in forming group image 132.This can be done by selecting the base image 140. Alternatively,controller 32 can cooperate with signal processor 26 to determine whichof the images available in the set of imaging information S has thehighest overall combined attribute ranking for the various elementsexamined. Alternatively, controller 32 can cooperate with signalprocessor 26 to determine which of the images available in set ofimaging information S requires the least number of image processingsteps to form a group image therefrom. For example, where the fewestnumber of image processing steps is the criterion for selecting the baseimage, then image 118 can be selected as only one step needs to beperformed the step of fixing the appearance of the smile attribute offace 120. Alternatively, the step of selecting an interim image cancomprise selecting the image that can most efficiently or most quicklybe improved. For example, in FIG. 5, image 114, requires processing ofboth eye elements 124 and 126, thus requiring more than one correction,however, the processing effort required to correct the appearance of theeye can be substantially less than the processing effort required toimprove the appearance of the mouth element 128 of face object 120 inimage 118. Other stratagems can also be used for selecting the interimimage.

The attributes of the interim image are then examined to determinewhether each of the elements of the objects in the template image hasattributes of the highest order for that attribute. Where an element isfound that does not have attributes of the highest order, thencontroller 32 and image processor 26 extract imaging information fromthe set of imaging information S that corresponds to the highest orderedappearance of that element and inserts that imaging information into theinterim image in place of the imaging information in the interim imageassociated with that element. In this way, a multi-element image isformed with each object in the image having elements with preferredattributes. Further, such attributes are based upon actual scene imaginginformation such as actual facial expressions and are not based uponimaging information manufactured during the editing process. Thisprovides a group image having a more natural and a more realisticappearance.

It will be appreciated that group photos and other group images can becaptured of scenes and circumstances wherein it is not preferred thateach member of the group smiles. Rather for certain group photos adifferent mood or facial expression can be preferred. A user of imagingsystem 10 can use user input system 34 to define such expressions. Forexample, a desired facial expression of “scared” can be selected by theuser of imaging system 10.

FIG. 6 shows how the selection of the desired facial expression or moodaffects the composite picture outcome. In this example, a base image 140is selected and elements are identified. In the illustration of FIG. 6,base image 140 is comprised of two elements: a first face 142 and asecond face 144. First face 142 has a neutral expression and second face144 has a neutral expression. To compose an image having the desiredscared expression, eye elements 146 and 148 and mouth elements 150 and152 are examined over the time period captured in the set of imaginginformation and ordered as described above. However, in this example,the ordering is done with eye elements 146 and 148 and mouth elements150 and 152 ordered in accordance with their similarity to eye elements164 and mouth element 166 associated with a scared expression template162 stored within a template library 154 that also contains otherexpression templates such as a happy template 156, an angry template 158and cynical template 160. The template library 154 can comprise, forexample, a library of template images or other imaging information suchas image analysis and processing algorithms that associate theattributes of elements in an object, such as a mouth or eye elements ina facial image, with an expression. The templates in library 154therefore can comprise images, algorithms, models and other data thatcan be used to evaluate the attributes of the elements detected in eachobject in the scene. The templates can be based on overall typicalanamorphic facial expressions or the templates can be derived based uponprevious photographs or other images of first face 142 and second face144. The templates can be stored in imaging system 10 or in a remotedevice that can be contacted by way of communication module 54.

As is shown in FIG. 6, after ordering, a group image 170 is thencomposed as is described above, with each object, e.g. face 142 and 144having scared eye elements 172 and 174, respectively and scared mouthelements 176 and 178 having an appearance associated with the highestordered attributes of the elements.

It will be appreciated that, in certain circumstances, the set ofimaging information S may not contain an expression that appropriatelyrepresents the desired expression or that does not suggest the desiredexpression to the extent desired. Accordingly, a threshold test canoptionally be used. For example, in the embodiment shown, in FIG. 6, theordering process can be performed so that the attributes of the featuresin the set of imaging information S are compared to scared template 162and ordered according to a scoring scale. When this is done, the overallscore for second face 144, for example, can be compared to a thresholdscore. Where the score is below the threshold, it can be determined thatthe set of imaging information S does not contain sufficient informationfor an appearance of the expression desired. When this occurs,controller 32 can use communication module 54 to obtain imaginginformation depicting the elements having desired attributes from aremote memory system 52 having a database or template library depictingsecond face 144. Controller 32 incorporated this remotely obtainedimaging information into group image 170 in order to more closely adaptthe appearance of eye elements 148 and mouth elements 152 of second face144 to the desired “scared” expression to yield the group image 170wherein second face 144 has the scared eye element 174 and scared mouthelement 178 that correspond to the desired appearance.

The selection of a desired expression can be made in a variety of ways.For example, the selection can be made on an image by image basis withthe selection made once for each image and applied to all elements inthe image. Alternatively, the selection of the desired expression can bemade on an element by element basis with each element having anindividually selected desired expression or other desired attribute. Forexample, certain persons may feel that their appearance is optimizedunder circumstances where they have a big smile while other persons mayfeel that their appearance is optimized with a more subdued expression.In such circumstances, desired expressions can be selected for eachperson in an image.

FIG. 7 shows an alternative embodiment of the present invention in whichat the ordering process is performed in a semi-automatic fashion. Inthis embodiment, a set of imaging information S is obtained (step 180).Controller 32 can obtain the set of imaging information to be sent bycapturing archival images or a set of archival imaging information suchas a video stream as is described above. Controller 32 can also obtain aset of images to be sent by extracting the digital images from a memory,such as a removable memory 48. A set of imaging information can also beobtained using communication module 54.

The set of imaging information S is provided to one or more decisionmakers (step 182). Controller 32 can provide the set of imaginginformation S to each decision maker such as for example a person whoseimage is incorporated into the set of imaging information S. This can bedone, for example, by presenting the set of imaging information S to theperson using display 30 or by using communication module 54 to transmitthe set of imaging information S to a remote terminal, personal digitalassistant, personal computer or other display device.

After the set of imaging information S has been provided to the decisionmakers, each decision maker reviews the set of imaging information andprovides an indication of which image in the set of imaging informationhas objects with elements having desired attributes (step 184). This canbe done in a variety of ways. For example, where an image includes agroup of elements, a decision can be made for each element in the set ofimaging information S as to which portion of the set of imaginginformation S depicts the element as having favored attributes. Forexample, one or more elements can be isolated for example byhighlighting the element in a base image and a decision maker can thenselect from the imaging information that portion of the imaginginformation that depicts that element as having favored attributes. Thisselection can be made using user input system 34 for example bydepressing the select-it button 68 shown in FIG. 2.

When a selection is made, user input system 34 generates a signal thatindicates which segment of the set of imaging information S has imaginginformation that depicts that person with elements having the attributespreferred by that person. Controller 32 detects the signals from userinput system 34 to indicate that the selected image contains desiredattributes. It will be appreciated that circumstances can arise wheremore than one decision maker makes recommendations as to which portionof a set of imaging information S contains a preferred attribute. Suchconflicts can be prevented by limiting certain decision makers toproviding input only on selected elements. For example, where a groupimage comprises an image of a group of people, each person in the imagecan act as a decision maker for the elements associated with that personbut not for others. Alternatively, such conflicts can be resolved byproviding each person in the image with a different group image tailoredto the preferences of that person. The user input information can beused to help form the group image in two ways. In one way a userpreference can be used in place of the ordering step described above.Alternatively, the ordering steps described above in previousembodiments can be used and the user preference information can be usedto adjust the ordering performed on the imaging information.

Controller 32 then forms a group image based at least in part upon theinput received from user input system 34 (step 186). There are a numberof ways that this can be done. For example, a single group image can beformed based upon the input from all of the decision makers.Alternatively, controller 32 can be used to monitor the inputs from eachdecision maker with the group image selected by each decision makerusing the input made by other decision makers to adjust the ordering ofattributes of the elements.

FIG. 8 shows an illustration of another embodiment of the method of thepresent invention. In this embodiment, an interim image 200 is generatedby the imaging system 10 as described above. Interim image 200 containsimaging information that depicts an object 202 which is shown in FIG. 8as a face object having a mouth element 204. In this embodiment the setof imaging information depicting the mouth element is extracted from theset of imaging information and incorporated into the interim image asmetadata 206 associated with interim image 200. The image and metadataare transmitted by imaging system 10 to a home unit 208 such as apersonal computer, personal digital assistant, or other device by way ofa communication network 210. In this embodiment, metadata 206 isincorporated into interim image 200 in a way that permits a user of thehome receiver to access metadata 206 using software such as imageediting software, image processing software or a conventional webbrowser. The home user receives interim image 200 and, if desired,indicates that the user wishes to change or consider the option forchanging the attributes of one of the elements. This can be done, forexample, by hovering a mouse cursor over mouth element 204 of faceobject 202 in interim image 200 or otherwise indicating that an area ofinterim image 200 contains an element.

When this occurs, home unit 208 extracts the set of imaging informationassociated with mouth element 204 from metadata 206 and provides imaginginformation based upon the set of imaging information from which thehome user can select attributes that are preferable to the home user. Inthe embodiment illustrated, when the home user indicates a desire tochange the appearance of mouth element 204, a slide bar 212 appears onhome unit 208. By sliding slide bar 212 the user can move through theavailable set of imaging information associated with that image andselect imaging information having preferred attributes. The homereceiver records an indication of which attributes are found to bepreferable by the home user and adjusts the image to include thoseattributes. This allows each person captured in an image to adjust theattributes for that person in the archival image in order to optimizetheir appearance. The adjusted group image can be adapted so that theadjusted group image and any copies of the image made from the adjustedgroup image will contain the preferred image attributes. In anotheralternative of this type, each recipient of the group image is providedwith a copy of the group image that contains metadata for each imageelement and can select attributes for each element to form a local groupimage that is customized to the preferences of the recipient.

Optionally, the home receiver also provides a feedback signal by way ofcommunication network 210 to imaging system 10 or some other device 214such as a storage device, server or printer containing the interim imagewith the feedback signal indicating adjustments made by home unit 208.This information can be received by imaging system 10 or other storagedevice 214 and then used to form an adjusted archival image having auser selected and optimized appearance. It will be appreciated that suchediting can be performed by user input system 34 to perform the functionof selecting desirable attributes for the adjusted archival image.

Although imaging system 10 has been shown generally in the form of adigital still or motion image camera type imaging system, it will beappreciated that imaging system 10 of the present invention can beincorporated into and the methods and computer program product describedherein can be used by any device that is capable of processing a set ofimaging information examples of which include: cellular telephones;personal digital assistants; hand held, tablet, desktop, notebook andother personal computers and image processing appliances such asinternet appliances and kiosks. Further, imaging system 10 can comprisea film or still image scanning system with lens system 23 and imagesensor 24 adapted to scan imaging information from a set of images on aphotographic film or prints and can even be adapted to obtain imageinformation from a set of film image negatives. In such an application,imaging system 10 can comprise for example a personal computer,workstation, or other general purpose computing system having such animaging system.

Alternatively, imaging system 10 can also comprise a scanning systemsuch as those employed in conventional photofinishing systems such asthe photographic processing apparatus described in commonly assignedU.S. Pat. No. 6,476,903 entitled “Image Processing” filed by Slater etal. on Jun. 21, 2000.

The invention has been described in detail with particular reference topreferred embodiments thereof, but it will be understood that variationsand modifications can be effected within the spirit and scope of theinvention.

PARTS LIST

-   10 imaging system-   20 body-   22 image capture system-   23 lens system-   24 image sensor-   26 signal processor-   28 display driver-   30 display-   32 controller-   34 user input system-   36 sensors-   38 viewfinder-   40 memory-   46 removable memory slot-   48 removable memory-   50 removable memory interface-   52 remote memory system-   54 communication module-   60 shutter trigger button-   62 “wide” angle zoom button-   64 “tele” zoom button-   66 fix-it button-   68 select-it button-   70 enter group image forming mode step-   72 obtain set of imaging information step-   74 distinguish objects and elements step-   76 examine attributes of elements step-   78 distinguish objects and elements over set of imaging information    step-   80 examine attributes of elements over set of imaging information    step-   82 order imaging information step-   84 compose image step-   90 base image-   92 face-   94 face-   96 eye element-   98 mouth element-   100 eye element-   102 mouth element-   110 set of imaging information-   112 image-   114 image-   116 image-   118 image-   120 face object-   122 face object-   124 eye element-   126 eye element-   128 mouth element-   130 mouth element-   132 group image-   140 base image-   142 face-   144 face-   146 eye element-   148 eye element-   150 mouth element-   152 mouth element-   154 template library-   156 happy template-   158 angry template-   160 cynical template-   162 scared template-   164 eye element-   166 mouth element-   170 group image-   172 scared eye element-   174 scared eye element-   176 scared mouth element-   178 scared mouth element-   180 obtain set of imaging information step-   182 provide set of imaging information to decision maker(s) step-   184 receive selection signal step-   186 form image based on selection signal step-   200 interim image-   202 face object-   204 mouth element-   206 metadata-   208 home unit-   210 communication network-   212 slide bar-   214 other device

1. A method for forming an image, the method comprising the steps of:obtaining a set of imaging information depicting a scene over a periodof time; providing a base image based upon the set of image information;identifying elements in the base image; ordering portions of the set ofimaging information depicting each of the elements according to anattribute of each element; selecting imaging information from the set ofimaging information depicting each element according to the ordering;and, forming an image based upon the set of imaging information with theimage incorporating the selected image information.
 2. The method ofclaim 1, wherein the step of providing a base image comprisesautomatically selecting an image from the set of image information. 3.The method of claim 1, wherein the step of providing a base imagecomprises receiving a manual input selecting the base image.
 4. Themethod of claim 1, wherein the step of ordering the portions of the setof imaging information depicting each of the elements according to anattribute of each element is based at least in part upon a user input.5. The method of claim 1, wherein the step of ordering the portions ofthe set of imaging information depicting each of the elements accordingto an attribute of each element is based at least in part upon userinputs from more than one user.
 6. The method of claim 1, wherein thestep of ordering the portions of the set of imaging informationdepicting each of the elements according to an attribute of each elementcomprises comparing the appearance of the imaging information depictingeach of the elements to a template over the period of time that theelement is depicted in the set of imaging information and ordering theimaging information associated with the element over the set of imaginginformation based upon the attributes for the element.
 7. A computerprogram product having data stored thereon for causing an imaging systemto perform a method for forming an image, the method comprising thesteps of: obtaining a set of imaging information depicting a scene overa period of time; providing a base image based upon the set of imageinformation; identifying elements in the base image; ordering theportions of the set of imaging information depicting each of theelements according to an attribute of each element; selecting imaginginformation from the set of imaging information depicting each elementaccording to the ordering; and, forming an image based upon the set ofimaging information with the image incorporating the selected imageinformation.
 8. An imaging system comprising: a source of a set of imageinformation; a signal processor adapted to obtain a set of imaginginformation depicting a scene over a period of time, to provide a baseimage based upon the set of imaging information to identify elements inthe base image, and to order the portions of the set of imaginginformation depicting each of the elements according to an attribute ofeach element, wherein the processor selects imaging information from theset of imaging information depicting each element according to theordering; and forms an archival image based upon the set of imaginginformation with the archival image incorporating the selected imageinformation.
 9. The imaging system of claim 8, wherein the signalprocessor automatically selects a base image from the set of imageinformation.
 10. The imaging system of claim 8, further comprising adisplay for presenting the set of imaging information and a user inputadapted to receive an input selecting a base image from the set of imageinformation.
 11. The imaging system of claim 8, further comprising auser input, wherein the signal processor orders the portions of the setof imaging information depicting each of the elements according to anattribute of each element based at least in part upon a user input. 12.The imaging system of claim 8, further comprising a user input, whereinthe signal processor orders the portions of the set of imaginginformation depicting each of the elements according to an attribute ofeach element based at least in part upon inputs from more than one user.13. The imaging system of claim 8, wherein the signal processor ordersthe portions of the set of imaging information depicting each of theelements according to an attribute of each element by comparing theappearance of the imaging information depicting each of the elements toa template over the period of time that the element is depicted in theset of imaging information and ordering the imaging informationassociated with the element over the set of imaging information basedupon the attributes for the element.